diseasedatahub.org/manifest.json

Deployment changes

• arborist
  • Replace Travis with a GitHub workflow for CI (Fix lint CI + replace travis with GH workflow arborist#150)
  • Needs Fence later than 4.22.2 or 2020.08 (which introduces the new JWT scopes claim). ((PXP-6617): remove scopes from jwt aud claim arborist#130)
  • Assuming any commons running Arborist < 2.3 was using user sync only as AuthZ provider (chore(authz): set authz_provider in migration arborist#111)
  • One-off: Update authz_provider to the correct values (Feat/authz provider arborist#109)
• fence
  • Deployment changes to enable this feature for commons are described here https://github.com/uc-cdis/fence/blob/e36f6574ade17bb8c9c0c9df8801f9a59de423b1/docs/fence_multifactor_authentication_guide.md (feat(DCF-1446): Assign MFA policy on login fence#1106)
  • Update fence-config.yaml to include parent_to_child_studies_mapping to define parent/child relationships within the dbGaP block of the config. (feat(PXP-10776): Cascading Authorization fence#1102)
  • Allow setting INDEXD_PASSWORD via environment variable. (allow setting INDEXD_PASSWORD via env var fence#1092)
  • Systems that use visas parsing must update their Fence configuration to set ENABLE_VISA_UPDATE_CRON to True and set GLOBAL_PARSE_VISAS_ON_LOGIN to either True or None (GPE-891 Add ENABLE_VISA_UPDATE_CRON setting fence#1087)
  • Requires a Fence DB migration (PXP-10541 Client credentials rotation fence#1068)
  • Configure the ALLOWED_DATA_UPLOAD_BUCKETS setting to allow users to upload to buckets other than DATA_UPLOAD_BUCKET (PXP-10361 Allow specifying the upload bucket fence#1051)
  • B/c the default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP, you MUST ensure that an instance that should comply with FedRAMP does NOT override this fence configuration (e.g. it should be 15 minutes) SESSION_TIMEOUT: 900 (feat(session): change default to 15 minutes [PPS-122] fence#1040)
  • Requires a Fence DB migration (Client.redirect_uri is now optional) (PXP-10113 Client credentials grant fence#1033)
  • Requires database migration for a new table to Fence, google Access expiration to add new expires column, and to create iss_sub_pair_to_user table Update manifest.json #968 updated BRAIN production data model to 1.0.0 #963 feat(stage-tiered-access): Updated portal versions to fix tiered acce… #973 (Feat/passport fence#964)
  • Requires Arborist >= 2022.04 OR >=3.4.0 (PXP-9480): Support merge param for PUT /resource endpoint arborist#143 (Feat/passport fence#964)
  • To enable Passports -> DRS in an environment you must update Fence Configuration to at a minimum specify GA4GH_PASSPORTS_TO_DRS_ENABLED: true (only do this after coordinating with the environment owner as this has several requirements before it can be enabled, including the need to use Indexd's authz field in ALL records rather than acl) (Feat/passport fence#964)
  • Requires a Fence DB migration (chore(migration): add authz column to project fence#1003)
  • Requires a database migration to create iss_sub_pair_to_user table ((PXP-8682): Add visa validation and sync for data download fence#973)
  • Google Access expiration requires a database migration to add new expires column! (feat(storage_expiration): add expires to db for Google access, create… fence#968)
  • Requires database migration. Adding a new table to Fence. ((PXP-8679): Add new db table and update login to add entries fence#963)
  • Requires these cloud-automation updates to revproxy: (PXP-7565): Exclude Fence register-user blueprint from CSRF check cloud-automation#1580 ((PXP-7565): User registration, plus CSRF changes fence#906)
  • By default Fence will allow only visas issued by RAS staging, RAS prod, and itself; if a different set of allowed visa issuers is needed, edit the GA4GH_VISA_ISSUER_ALLOWLIST config value. Otherwise, no action needed. ((PXP-6339): RAS visa validation fence#933)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence (You have to add any valid redirects to the client before deploying this) ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • For existing deployments with "ENABLE_AUDIT_LOGS" turned ON, to DISABLE pushing audit logs to an AWS SQS, the configuration "PUSH_AUDIT_LOGS_CONFIG.type" must be changed to "api" (default: "aws_sqs"). If audit-service has never been deployed, no action required. (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • To ENABLE pushing audit logs to an AWS SQS (REQUIRES audit-service 1.0.0 or more recent), create/update the configuration file's "PUSH_AUDIT_LOGS_CONFIG" field ("type: aws_sqs" (default), and the SQS URL and region can be copied from the audit-service configuration - or run gen3 sqs info $(gen3 api safe-name audit-sqs)), see https://github.com/uc-cdis/fence/blob/fec3dd937d68384e86b2ed6f9b53daf0026c2df9/fence/config-default.yaml#L632-L636. Then run kubectl delete secret fence-config and gen3 kube-setup-fence (PXP-7805 Push audit logs to an AWS SQS fence#923)
  • This commit includes patches to allow old style refresh tokens and API keys to pass new style validation, while issuing new style refresh tokens and API keys. Since refresh tokens and API keys have a default TTL of 30 days, stay on this tag for 30 days if possible. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Old access tokens will NOT work once this is deployed so active sessions may be temporarily interrupted. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Audit logs for user logins and data downloads can be enabled by deploying the audit-service and adding setting ENABLE_AUDIT_LOGS to the fence config (see https://github.com/uc-cdis/fence/blob/1c850d07e54f8527dbde182fdd9dd593d18f660e/fence/config-default.yaml#L586-L588) (PXP-7447 audit-service integration fence#885)
  • REQUIRES A FENCE DB MIGRATION, since with this update there is now an additional column in the authorization_code table "refresh_token_expires_in" (Feat/refresh expires in fence#848)
  • Commons configured with RAS staging login must add the "discovery_url" fence config, because it now defaults to RAS prod (Fix/ras url fence#824)
  • If the fence config already has [CLIENT_, USER_, SESSION_]ALLOWED_SCOPES fields, and GA4GH AAI integration is desired, then add ga4gh_passport_v1 to all those lists. If default config values are being used then no change required. (PXP-6406 Feat/ga4gh scope fence#799)
  • To use the new Cognito integration, apply in fence config the changes made to config-default.yaml. Otherwise no action required. (PXP-5613 Feat/cognito fence#767)
  • In fence-config, the region field of S3_BUCKETS is now required: if not specified, Fence will print a warning and try to call GetBucketLocation. Fence will error at startup if that doesn't work. (Fix/make region required for s3 bucket config fence#749)
• guppy
  • We need to upgrade the target environment to use ElasticSearch 7.10 or OpenSearch before using this new guppy. (Feat/es7 guppy#200)
  • We also need to upgrade Tube to new version that is compatible with ES7. Checking this link for the corresponding PR of Tube. (Feat/es7 guppy#200)
  • update node from 14 to 16 (update node to 16 guppy#219)
  • To deploy SearchFilters in a tab of the Explorer page, add an entry to data{/file}ExplorerConfig.filters.tabs.searchFields. NOTE: Do not add the same field to both searchFields and fields, this will cause strange behavior in the search field. Example config: ((PXP-6359) Add Search Filter guppy#99)
  • "filters": { ((PXP-6359) Add Search Filter guppy#99)
  • "tabs": [ (https://github.com/(PXP-6359) Add Search Filter guppy#99)
  • { ((PXP-6359) Add Search Filter guppy#99)
  • "title": "Subject", ((PXP-6359) Add Search Filter guppy#99)
  • "searchFields": [ (https://github.com/(PXP-6359) Add Search Filter guppy#99)
  • "subject_id", ((PXP-6359) Add Search Filter guppy#99)
  • "submitter_id", ((PXP-6359) Add Search Filter guppy#99)
  • "consent_codes" ((PXP-6359) Add Search Filter guppy#99)
  • ], ((PXP-6359) Add Search Filter guppy#99)
  • "fields": [ (https://github.com/(PXP-6359) Add Search Filter guppy#99)
  • "data_type", ((PXP-6359) Add Search Filter guppy#99)
  • "data_format", ((PXP-6359) Add Search Filter guppy#99)
  • ... ((PXP-6359) Add Search Filter guppy#99)
  • ] ((PXP-6359) Add Search Filter guppy#99)
  • } ((PXP-6359) Add Search Filter guppy#99)
  • Increased node.js max header size to prevent issue with large NIH login session cookies hitting request header limit. ((PXP-5570) Bump header size limit to 16kb guppy#79)
• hatchery
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PR fix(dataguids.org): add manifest.json #35) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add gen3-volume-location to the Hatchery configuration file for each container to overwrite the directory that API key will be saved to (PR fix(account): portal to master #33) (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option ecs for enabling ECS decoupled workspace (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Update the Hatchery configuration file pay-models block to add option subnet for setup transit gateway subnet addresses (PXP-8779 HP-347 Feat/ecs hatchery#38)
  • Add pay-models-dynamodb-table to the Hatchery configuration file to fetch pay model data from DynamoDB (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
  • Update the Hatchery configuration file pay-models block to change the user field name to user_id (PXP-8341 Fetch pay models from DynamoDB hatchery#35)
• indexd
  • New columns added to index_record table. Alembic migration needed. (feat(DCF-450): Add DRS Fields indexd#357)
  • Include example secrets to load PostgreSQL settings for indexd (Add files for psql ssl support indexd#317)
  • When running gen3authz, backoff package needed separate poetry run pip install (PXP-6824: return 401 errors from Arborist indexd#304)
  • Indexd's blank record endpoints now allow direct iteration from a user with their access token but requires the file upload arborist role to be modified so the service is not just fence but instead * so indexd works too. see https://github.com/uc-cdis/commons-users/pull/908/files for example (Update blueprint.py indexd#275)
• manifestservice
  • ManifestService does not expect aws_access_key_id and aws_secret_access_key settings anymore. Instead, the service relies on AWS EKS IRSA credentials mounted to the pod. See https://github.com/uc-cdis/manifestservice/blob/master/docs/credentials_breaking_change.md for more details (feat(IAM-SA): Removed AKID references to use IAM SA role instead manifestservice#32)
• peregrine
  • The pidgin service can be removed if data-portal is on version 5.4.2/2023.05 or newer. Its functionalities are available at peregrine's /coremetadata endpoint (PXP-10708 Add coremetadata endpoint peregrine#198)
  • ASK AN ADMINISTRATOR TO DELETE THE PIDGIN DEPLOYMENT if you remove pidgin. This is NOT automatic (PXP-10708 Add coremetadata endpoint peregrine#198)
  • Peregrine does not rely on configuration files in the cloud-automation repository anymore. The Peregrine configuration files are in the Peregrine repository. (PXP-10163 Python 3.9 peregrine#191)
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) peregrine#142)
  • Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist peregrine#138)
  • Requires Arborist to be deployed (Feat/arborist peregrine#138)
  • look for INDEX_CLIENT instead of SIGNPOST in app.config (Remove signpost peregrine#135)
  • get INDEX_CLIENT_HOST env var instead of SIGNPOST_HOST (Remove signpost peregrine#135)
• sheepdog
  • Sheepdog does not rely on configuration files in the cloud-automation repository anymore. The Sheepdog configuration files are in the Sheepdog repository. (PXP-6245 PXP-9633 Poetry + Python 3.9 sheepdog#387)
  • The environment's user.yaml needs to be updated to add the new Sheepdog admin policy/resources/roles; for each user that has admin: true, grant them the new policy. See https://github.com/uc-cdis/commons-users/pull/810 for an example. (PXP-4102 Feat/arb admin sheepdog#319)
  • Fix Travis coverage reports (combine the results of all 3 tests into one coverage report). (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • ARBORIST_URL environment variable replaces broken default ARBORIST config variable (fix(arborist-url) sheepdog#304)
  • Optional configuration variable ARBORIST (arborist base URL) (Feat/arborist2 sheepdog#298)
  • Requires Arborist to be deployed (Feat/arborist2 sheepdog#298)
• tube
  • Requires pelican-export >= 0.5.x (fix(project_id): ensure project_id exists tube#131)
  • Breaking change adding underscore to all {subject,case,file}_id fields. Requires migrating gitops.json and etlMapping.yaml following these instructions: 1. Make a list of all the doc_type fields in etlMapping.yaml. 2. Change any instance of {doc_type}_id to _{doc_type}_id in gitops.json and etlMapping.yaml. (E.g. if doc_type is subject, replace all instances of subject_id in etlMapping.yaml and gitops.json with _subject_id). 3. If there is no change to the etlMapping in the PR, manually add a comment to the etlMapping or change the order of two props in order to force gitops-sync to run the ETL. (fix(project_id): ensure project_id exists tube#131)
• wts
  • This version of WTS requires a DB migration (in cloud-automation deployments, the DB should be migrated automatically) (PXP-9499 Encrypt refresh tokens workspace-token-service#40)

Breaking changes

• arborist
  • The GET /auth/mapping endpoint does not accept username as a query parameter anymore. It only supports parsing the username from the provided JWT. To specify a username, use the POST /auth/mapping endpoint. (PXP-10229 Client auth mapping arborist#153)
  • In step with Fence changes to issued JWTs, in which scopes are moved into a dedicated scope claim (Fence 4.22.2/2020.08) and out of the aud claim (Fence 5.0.0/2021.07), this commit changes Arborist JWT validation logic so that it does the same validation it was doing on the aud field, but does it on the scope field instead. This Arborist version must therefore be paired with a Fence later than 4.22.2/2020.08. Conversely, Fence 5.0.0/2021.07 will require Arborist later than this commit. ((PXP-6617): remove scopes from jwt aud claim arborist#130)
• fence
  • Removed most Prometheus support (feat(google_update): updated Flask and deps, also don't fail early with Google errors PXP-10900 fence#1086)
  • The default value for the GLOBAL_PARSE_VISAS_ON_LOGIN setting is now False instead of None. By default, parsing visas on login is now disabled and the parse_visas parameter cannot be used (GPE-891 Add ENABLE_VISA_UPDATE_CRON setting fence#1087)
  • Default session expiration changed from 30 minutes to 15 minutes to comply with FedRAMP (feat(session): change default to 15 minutes [PPS-122] fence#1040)
  • URL Signing when no_force_sign query param is provided: Previously Fence would make a decision based off if the data was public and no_force_sign provided. Fence will now NEVER sign if no_force_sign is provided (since the concept of "public" data has been abstracted out of Fence. Data access - public or not - is the sole responsibility of the policy engine). In other words, if no_force_sign is provided at the API level, Fence will respect that regardless of whether the resulting URL will actually work. The default Fence behavior should remain the same. chore(jupyter-nde1.1.0) #988 (Feat/passport fence#964)
  • URL Signing when no_force_sign query param is provided: Previously Fence would make a decision based off if the data was public and no_force_sign provided. Fence will now NEVER sign if no_force_sign is provided (since the concept of "public" data has been abstracted out of Fence. Data access - public or not - is the sole responsibility of the policy engine). In other words, if no_force_sign is provided at the API level, Fence will respect that regardless of whether the resulting URL will actually work. The default Fence behavior should remain the same. (feat(passport-exp): handle expiration edge cases [PXP-9235] fence#988)
  • /link/google will DENY requests with "redirect" param values that aren't in the "allowed redirects" for OAuth2/OIDC Clients in Fence ((PXP-8286): fix(google): add redirect validation to /link/google fence#927)
  • Remove scopes from "aud" claim in Fence-issued tokens. This means that in general, old tokens (minted by previous Fence releases) will not pass validation by this Fence. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Access tokens will still include scopes for compatibility with other microservices, but scopes are no longer guaranteed and should no longer be expected in the aud claim from this version on. ID tokens, refresh tokens, and API keys will not have scopes in the aud claim. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add custom "scope" claim in tokens and put scopes there instead. Custom validation for scopes claim added in Authutils 6.0.1 (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Add issuer to "aud" claim (alongside other aud values--so when there is a Fence client involved, this means the aud claim will include both Fence's BASE_URL and the client_id) (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • Remove custom "aud" claim validation and use normal validation instead (by bumping to Authutils 6.0.1). This means checking that the aud claim contains a value that identifies the consuming service, i.e. Fence's BASE_URL. (PXP-6617 Remove scopes from aud claim in tokens fence#839)
  • This Nginx status endpoint is a hard dependency for the nginx-prometheus-exporter side car. Any older fence image that is used with the Kubernetes deployment descriptor that is declared with the sidecar below will fail to come up. (chore(monitoring): Adopt new pybase img version. fence#817)
  • name: nginx-prometheus-exporter (chore(monitoring): Adopt new pybase img version. fence#817)
  • image: registry.hub.docker.com/nginx/nginx-prometheus-exporter:0.8.0 (chore(monitoring): Adopt new pybase img version. fence#817)
  • args: ["-nginx.scrape-uri", "http://127.0.0.1/nginx_status"] (chore(monitoring): Adopt new pybase img version. fence#817)
  • ports: (chore(monitoring): Adopt new pybase img version. fence#817)
  • containerPort: 9113 (chore(monitoring): Adopt new pybase img version. fence#817)
  • Usersync will no longer populate user's email field in Fence db with the user's username, UNLESS the username is an email address, in which case previous behavior applies (email field is populated with the username). In other words, nothing should change except in cases where usersync was putting invalid emails into user email fields anyway. (PXP-2802 feat(usync-emails) fence#788)
• guppy
  • Any environment using ElasticSearch 6.x will break with the new version (Feat/es7 guppy#200)
  • User can only see/query the data for a project if they have the following access: method read (or * - all methods) to service guppy (or * - all services) on that project resource. (previously they can see/query all the data if you any types of access to them) (PXP-6412 Feat/auth mapping guppy#93)
• hatchery
  • /launch endpoint throws an error when there is no pay model selected (For commons with paymodels) (Hatchery pay model improvements hatchery#70)
  • /setPaymodel throws an error when there is no active workspace running. (For commons with paymodels) (Hatchery pay model improvements hatchery#70)
  • In pay model configs, name has been changed to workspace_type (HP-682 Feat/multiple paymodels  hatchery#44)
  • In pay model configs, aws_account_id has been changed to account_id (HP-682 Feat/multiple paymodels  hatchery#44)
  • In pay model configs, ecs has its type changed from string to bool (HP-682 Feat/multiple paymodels  hatchery#44)
  • In pay model configs, vpcid has been removed (HP-682 Feat/multiple paymodels  hatchery#44)
• indexd
  • DRS responses for update_time and created_time reflect the content's updated and created times, not the time they were indexed or their index record was updated. (feat(DCF-450): Add DRS Fields indexd#357)
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (Revert "Revert "fix(migrate): improve acl->authz migration, remove de… indexd#338)
  • Ensure you are using Fence>=2021.10. Remove deprecated GA4GH access endpoints. these have existed (and been used) from the Fence microservice since last year. See cloud automation change here (fix(migrate): improve acl->authz migration, remove deprecated endpoints indexd#336)
  • indexd blank record endpoints no longer require uploader field (fix(blank): uploader is no longer technically required since arborist… indexd#277)
• peregrine
  • Python 3 Support (not expected to be backwards compatible) (Update to Python3  peregrine#154)
  • Use Arborist + gen3authz for auth requests (Feat/arborist peregrine#138)
• sheepdog
  • Sheepdog now validates formats. If your dictionary uses formats and you have existing data or submission pipelines that do not respect these formats, this might be a breaking change (PXP-9192 Validate format sheepdog#363)
  • Note that this PR will cause Sheepdog behavior to change slightly during a data submission flow in which the acl and/or authz field are previously populated in the corresponding Indexd record and the Indexd uploader field is not null and the Sheepdog submitter does not match the Indexd uploader. While the existing behavior will cause data submission to fail and report an error message to the user, this PR’s changes will mean that data submission can still succeed in this case of populated acl/authz and uploader mismatch. I believe this change in behavior is not problematic in that it would be unexpected in any commons for uploader and (acl and/or authz) to both be populated. ((PXP-6580): fix data submission by only writing acl and authz iff both are empty sheepdog#330)
  • Having admin: true in user.yaml no longer allows a user to do program/project CRUD in Sheepdog. Users now need to have the Sheepdog admin policy in Arborist. (PXP-4102 Feat/arb admin sheepdog#319)
  • Requires Python 3 (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • Updates Sheepdog to use new Indexd aliases endpoint rather than deprecated aliases endpoint (Feat/index record alias api indexd#248) (PXP-3408 Update Sheepdog to Python3 sheepdog#314)
  • Use Arborist + gen3authz for auth requests (Feat/arborist2 sheepdog#298)
• tube
  • Will break with current ES6 cluster (fix(es7): test tube with es-7 tube#238)
  • python 3! (Chore/py3 tube#119)